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Abstract 


The research literature on evidence-based academic interventions focuses predominantly on measures 
of intervention effectiveness. Intervention efficiency measures, in contrast, include the dimension of time 
required to achieve a level of effectiveness. The purpose of this study was to review the research literature 
for single-case design studies that included measures of both effectiveness and efficiency for academic 
(i.e., reading, math, writing, and spelling) interventions in schools. The study reviewed single-case design 
research on academic interventions published in six dominant peer-reviewed journals in school psychology 
between 2003-2013: School Psychology Review, School Psychology Quarterly, Psychology in the Schools, 
Journal of School Psychology, Journal of Behavioral Education, and the Journal of Evidence-Based 
Practices for Schools. The results of this study suggest that intervention efficiency measures are largely 
absent from the academic intervention research. The implications of this study are that both effectiveness 
and efficiency measures need to be considered as they each provide a unique contribution to determining 
the impact of an academic intervention. 

Key words: intervention efficiency, single-case designs. 


Introduction 


The field of school psychology has advanced from its original focus on the assessment of 
student deficits to its current emphasis on prevention and intervention through comprehensive 
service delivery aimed at the student- and systems-levels (National Association of School Psycholo- 
gists, 2010; Ysseldyke et al., 2006). The pursuit of positive, measurable outcomes for students and 
families relies on a foundation of empirically-supported interventions implemented in school set- 
tings (Villarreal et al., 2013). Although research has identified numerous academic interventions 
for use with students in schools (see Shapiro, 2004; Shinn, Walker, & Stoner, 2002), research on 
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evidence-based instructional and intervention procedures in school psychology journals remains 
relatively scarce (Bramlett et al., 2010; Villareal et al., 2013). 

The effects of academic interventions in applied settings can be investigated through the use 
of an internally-valid single-subject design where each participant serves as his/her own control 
(Kazdin, 2010; Kennedy, 2005). Single-case designs, developed to understand patterns of individual 
behavior in response to an independent variable (the intervention condition), are ideally suited to 
measure intervention effectiveness for a variety of targeted skills/behaviors (Barnett et al., 2004). 
When used in school settings, each student’s performance during an intervention (treatment) phase 
is compared to the student's baseline performance prior to intervention. Multiple-baseline designs 
incorporate within-student and between-student comparisons through the use of an intervention 
implemented individually across several students (or settings, or targeted skills) with a systemati- 
cally delayed introduction. Alternating-treatment designs enable the direct comparison of academic 
interventions on student performance. Single-case designs provide the experimental rigor needed 
to establish internal validity through the detailed description of the intervention condition and the 
precise measure of degree to which the intervention was implemented as planned (i.e., procedural 
integrity, treatment integrity, intervention adherence). As such, single-case design methodology 
is well-suited as a foundation for intervention research examining the key features of academic 
interventions that lead to positive outcomes for students and the effects of these interventions when 
they are modified to meet the demands (i.e., time resources, and materials) of the natural school 
environment (Hawkins, Morrison & Musti-Rao, 2008). 


Problem of Research 


Despite the critical need to establish a body of intervention research with empirical support 
either through scholarly journals or a recognized expert-reviewed database (Hoagwood & John- 
son, 2003; Kratochwill & Shernoff, 2004), there has been a well-documented lack of intervention 
research studies in the top school psychology journals (Bramlett et al., 2010, Harrison, 2000; Kra- 
tochwill & Stoiber, 2000; Robinson, Skinner, & Brown, 1998; Seethaler & Fuchs, 2005; Shriver & 
Watson, 2005; Strein, Cramer, & Lawser, 2003; Villarreal et al., 2013). For example, in their review 
of articles published by the four major school psychology journals (i.e., Journal of School Psychol- 
ogy, Psychology in the Schools, School Psychology Quarterly, and School Psychology Review) over the 
11-year period from 1995 to 2005, Bramlett and associates (2010) found only 16 articles focused 
on reading interventions, 10 articles examined math interventions, and only one study explored a 
writing intervention. Of these, single-case designs were used in 11 (69%) of the reading intervention 
articles, 3 (30%) of the math intervention articles, and the 1 writing intervention article. Likewise, 
Bliss, Skinner, Hautau, and Carroll (2008) reviewed articles published in the same four major 
school psychology journals from 2000-2005 and identified 42 (4.5% of the 929 studies reviewed) 
intervention studies that used single-case designs for academic and social/behavioral interventions. 

In the most recent review of intervention research articles published in six school psychol- 
ogy journals (i.e., Journal of Applied School Psychology, Journal of School Psychology, Psychology in 
the Schools, School Psychology International, School Psychology Quarterly, and School Psychology 
Review) from 2005 to 2009, Villarreal and associates (2013) found that the percentage of interven- 
tion studies was higher (9.3%) than had been previous reported in studies that focused only on 
the top four school psychology journals. Among the intervention research articles, 29.5% of the 
studies employed a single-case design methodology. The current study will extend the previous 
research by examining the use of single-case designs in the intervention research published in six 
journals relevant to school psychology with attention to the use of both effectiveness and efficiency 
outcome measures. 


Research Focus 


The goal of intervention research studies is to establish confidence that (a) within the con- 
text of the study the intervention caused the desired change in behavior, (b) the intervention or 
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a similar intervention could have a similar effect across students, settings, time, and target skills/ 
behaviors, and (c) the intervention is practical, allowing others to implement it with success across 
contexts (Skinner & Skinner, 2007). Given the goal of intervention research, intervention research- 
ers have argued that intervention effectiveness is not sufficient for understanding the impact of 
an intervention due to important practical considerations relevant in schools (i.e., accountability 
for instructional time, resource allocation) (Cates, Burns, & Joseph, 2010; Skinner, 2008). As such, 
intervention efficiency, the amount of change over baseline performance as well as an evaluation 
of the instructional time (i.e., number of minutes) required to bring about that change, needs to 
be reported along with intervention effectiveness (Cates et al., 2003; Skinner, 2008; Skinner et al., 
1997; Skinner, Belfiore, & Watson, 1995). 

To illustrate, in a study comparing the effects of the Detect, Practice, and Repair (DPR) in- 
tervention package versus traditional Cover, Copy, and Compare (CCC) procedures to increase 
multiplication math fact accuracy (i.e., number and percentage of math facts correct) and fluency 
(i.e, number of math facts correct per 2 minutes) among four Grade 4 students, both the CCC 
intervention and the DPR intervention were effective at bringing about changes in math fact flu- 
ency and math fact accuracy relative to baseline performance (Rahschulte et al., 2014). DPR trend 
estimation lines were steeper than CCC trend estimation lines for all students for both math fact 
fluency and math fact accuracy, suggesting that the DPR intervention package was more effective 
than CCC procedures at bringing about changes in math fact fluency and math fact accuracy using 
an alternating treatments design with a modified control condition. In comparing the interven- 
tion efficiency of CCC versus DPR, the comparison of learning rates indicated that DPR was more 
efficient than CCC for math fact fluency for three of the four student participants, whereas CCC 
was more efficient than DPR for math fact accuracy. 

Intervention efficiency is critically important for research and practice. Focusing exclusively on 
intervention effectiveness in research could yield misleading results (Skinner, 2008). For practition- 
ers in school settings, intervention selection necessarily involves the consideration of interventions 
procedures that will likely produce the greatest academic growth in the shortest amount of time. 
Intervention effectiveness and efficiency both need to be reported in the intervention research so 
that researchers and practitioners can make sound decision regarding the impact of an interven- 
tion (Skinner, 2008). 

The current study will extend the previous research by exploring the inclusion of intervention 
effectiveness and efficiency outcomes in the academic intervention research in six journals relevant 
to school psychology. In their review of the four major school psychology journals from 1995 to 
2005, Bramlett and associates (2010) identified only two studies that directly examined the length 
of each intervention session when determining intervention success. 


Methodology of Research 
General Background of Research 


Single-case designs are commonly used to measure internal validity in intervention research. 
Multiple-baseline designs, alternating treatment designs and ABAB designs provide evidence 
regarding the effect of the intervention (independent variable) on the target variable (dependent 
variable) while controlling for extraneous variables. Although the effect of the intervention on the 
target variable is typically presented in terms of its effectiveness, the efficiency of the intervention is 
an important variable that is often overlooked. The purpose of this study was to review the research 
literature for single-case design studies that included measures of both effectiveness and efficiency 
for academic (i.e., reading, math, writing, and spelling) interventions in schools. 


Sample 


For the purposes of the current review, a literature search was conducted using the PsychInfo 
and ERIC databases. The search was limited to peer-reviewed, empirical research studies published 
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between 2003 and 2013 in order to extend and expand the literature search conducted by Bramlett 
et al., (2010). As in the Bramlett et al. (2010) study, four school psychology journals were reviewed: 
Psychology in the Schools (PITS), School Psychology Quarterly (SPQ), School Psychology Review 
(SPR), and The Journal of School Psychology (JSP). The Journal of Behavioral Education (JOBE) 
and the Journal of Evidence-Based Practices for Schools (JEBP) were also reviewed in the current 
study. Given that the focus of this study was on the use of effectiveness and efficiency measures 
for academic interventions (i.e., reading, writing, or math), articles targeting behavioral variables 
were not included. 

Only studies of school-age children in general or special education programs were included in 
this review. All of the studies were required to (a) include an internally-valid single-case design, (b) 
report observational data, (c) provide checks for reliability, and (d) take place in a school setting. 

A total of 2,739 references were identified for articles published among the six journals between 
2003-2013. After removing articles that did not involve an empirical research study of an academic 
intervention, 151 studies remained. The abstracts were reviewed in order to identify studies meeting 
the selection criteria, and if that information was insufficient, the full article was reviewed. At this 
point, an additional 67 studies were deleted due to a lack of fit with the inclusion criteria. In the 
end, a total of 83 studies were identified that met all inclusion criteria (a complete list of references 
is available upon request from Julie Q. Morrison). 


Instrument and Rating Procedures 


Using the university library’s e-journal database, four coders (one school psychology fac- 
ulty member, one psychology faculty member, and two school psychology graduate students) 
were trained to locate, retrieve, and code the articles identified in the selected school psychology 
journals. The coding conventions were developed jointly during a one hour face-to-face meeting. 
The resulting coding protocol was comprised of 11 key variables, which included: journal name, 
publication year, citation, participant (e.g., gender, age, and disability), setting, number of students 
targeted, length of intervention condition, duration of intervention per session, effectiveness vari- 
able, efficiency variable, and single-case research design. The primary variables of interest were the 
frequency of articles with (a) results reported without reference to the amount of time required to 
yield the outcome (effectiveness variable), and (b) results reported as a function of the amount of 
time required to implement the intervention (efficiency variable). 

In coding the efficiency variable, a distinction was made between an efficiency variable (i.e., 
learning rate) and an effectiveness variable reported as a rate of improvement. Intervention efh- 
ciency is computed by dividing the number of units learned by the amount of instructional time 
(i.e., number of minutes) (Cates, Burns & Joseph, 2010; Skinner, Belfiore & Watson, 1995). In con- 
trast, a rate of improvement is calculated by dividing the number of units learned by the amount 
of time lapsed (i.e., weeks of school). Whereas a rate of improvement provides useful information 
regarding the effectiveness of an intervention as evident in weekly gains in student skill fluency, to 
be considered an efficiency variable, the gains in student skill fluency need to be anchored by the 
amount of instructional/intervention time. 

To established inter-rater reliability, one half of the articles were randomly selected for review 
by a second coder. The two researchers in this study also served as a second coder for any of the 
articles identified as having an efficiency variable in the first review that were not selected for inclu- 
sion in the second round of reviews. As such, 56.5% of the articles were reviewed by an independ- 
ent second coder. The second review focused on the primary variables of interest: (a) evidence of 
an effectiveness variable, and (b) evidence of an efficiency variable. The percentage of inter-rater 
reliability was calculated by dividing the number of agreements by the number of agreements 
and disagreements and multiplying by 100. Inter-rater reliability for the effectiveness variable(s) 
was 100%. Inter-rater reliability for the efficiency variable was 89.6%. Any questions, concerns, 
or disagreements were resolved by the primary author in consultation with the secondary author. 
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Data Analysis 


Descriptive statistics were used to summarize and analyze the data gathered with the coding 
protocol. Specifically, frequency counts, cross-tabulations, and percentages were calculated in the 
analysis of publication outcomes by year and publication outcomes by journal. 


Results of Research 


Academic Intervention Articles by Journal 


Over the 11-year period, 2,739 articles were published in the six journals relevant to school 
psychology in the years 2003 through 2013 (See Table 1). Of these articles, only 3.1% overall focused 
on the effectiveness of an academic intervention using a single-case design. The greatest number of 
articles examining the impact of academic interventions using single-case designs were found in 
the two journals not included in the previous studies of the intervention research literature (Bliss 
et al., 2008; Bramlett et al., 2010; Villarreal et al., 2013), the Journal of Evidence Based Practices in 
Schools and the Journal of Behavioral Education. The Journal of Evidence Based Practices in Schools 
published 17 articles (14.2% of the articles published) dedicated to academic interventions using 
single-case designs. The 17 articles explored academic interventions targeting reading/early lit- 
eracy (10), math (5), spelling/vocabulary (1), and writing (1). The Journal of Behavioral Education 
published 43 articles (18.4% of the articles published) focused on academic interventions using 
single-case designs. The 43 articles examined academic interventions targeting reading/early literacy 
(25), math (9), spelling (4), and writing (5). Among the four major school psychology journals, 
Psychology in the Schools had the highest frequency of articles examining academic interventions 
using single-case designs with 14 articles, but this only represented 1.6% of the total number of 
articles published by Psychology in the Schools. The 14 articles focused on academic interventions 
targeting reading/early literacy (9) and math (5). The remaining three school psychology journals 
published very few articles investigating the effectiveness of academic interventions using single- 
case designs: Journal of School Psychology (1 reading intervention article, representing 0.1%), School 
Psychology Quarterly (1 writing intervention article, representing 0.3%), School Psychology Review 
(9 articles: 7 reading, 2 math, representing 2.0%). 


Table 1. Frequency and percentage of articles examining academic 
intervention effectiveness using single-case designs in journals 
relevant to school psychology, 2003-2013. 


Number (and %) of 


Total Number of Academic Intervention 


Journal Name Articles Published : : : 
Articles using —_Single- 
2003-2013 ; 
Case Designs 
Journal of Evidence Based Practices in Schools 120 47 (14.2%) 
Journal of Behavioral Education 234 43 (18.4%) 
Journal of School Psychology 738 4 (0.1%) 
Psychology in the Schools 894 14 (1.6%) 
School Psychology Quarterly 309 1 (0.3%) 
School Psychology Review 444 9 (2.0%) 


Total 2,739 85 (3.1%) 
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Effectiveness and Efficiency Variables by Journal 


Of the 85 empirical research articles examining academic interventions using a single-case 
design, 85 (100%) featured an effectiveness variable(s), but only 14 (16.5%) included an efficiency 
variable. Efficiency variables were most frequently identified in the Journal of Behavioral Education 
(6), followed by Psychology in the Schools (4), and School Psychology Review (3) (See Figure 1). The 
Journal of Evidence Based Practices had only one article that included an efficiency variables and 
the Journal of School Psychology and School Psychology Quarterly did not have any studies published 
with efficiency variables reported. 
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Figure 1. Frequency of studies reporting academic effectiveness and efficiency 
outcomes by journal, 2003-2013. 


Effectiveness and Efficiency Variables by Publication Year 


The publication of research articles examining the effectiveness of academic interventions 
using a single-case design was variable across the 11-year period from 2003 through 2013. Studies 
investigating the effectiveness of academic interventions were published most frequently in 2006 and 
2010. There has been a steady downward trend in the frequency of published effectiveness studies 
from 2010 to 2013 (See Figure 2). With the exception of 2007 in which four studies included an 
efficiency variable, no more than two articles featured an efficiency variable in a given year. 
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Figure 2. Frequency of studies reporting academic effectiveness and efficiency 
outcomes by publication year. 
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Discussion 


The aim of the present literature review was to update and extend the previous studies of the 
intervention research literature found in school psychology journals, with a particular focus on the 
presence of academic effectiveness and efficiency outcome measures. Our literature review yielded 
some important information regarding the publication trends over the 11-year period from 2003 
through 2013. The review demonstrated that two journals relevant to school psychology, but not 
considered to be among the top four school psychology journals, published the greatest number of 
articles dedicated to academic intervention research studies using single-case designs. The journals 
relevant to school psychology, Journal of Behavioral Education and the Journal of Evidence Based 
Practices in Schools, were not included in previous studies of the intervention research (Bliss et al., 
2008; Bramlett et al., 2010; Villarreal et al., 2013), and yet clearly make a much needed contribution 
to advancing the body of intervention research with empirical support. 

Consistent with previous findings, an examination of the top four school psychology journals 
(and collectively all six journals in this literature review) yielded relatively few articles describing 
the results of academic interventions using a single-case design. This trend persists over an 11- 
year period, despite long-standing calls to address the scarcity of intervention research within the 
school psychology literature (Bramlett et al., 2010, Harrison, 2000; Hoagwood & Johnson, 2003; 
Robinson, Skinner & Brown, 1998; Seethaler & Fuchs, 2005; Shriver & Watson, 2005; Strein, Cramer 
& Lawser, 2003; Villarreal et al., 2013). In the present study, 3.1% of the articles published in the 
six journals relevant to school psychology focused on academic interventions using a single-case 
design. This figure is similar to the results obtained by Bliss and associates (2008), who reported 
4.5% of the articles published in the four major school psychology journals from 2000 to 2005 
described intervention studies that used single-case designs for academic and social/behavioral 
interventions. Likewise, Villarreal and associates (2013) found 9.3% of the articles published in six 
school psychology journals from 2005 to 2009, were dedicated to academic and social/behavioral 
intervention research, with slightly less than a third of these articles employing a single-case design. 
The scarcity of academic intervention research available to school psychologists in the most promi- 
nent journals in the field is troubling given the reliance of practitioners on the empirical studies 
to make informed decisions regarding the use of evidence-based interventions. This finding also 
has implications for researchers who may forgo scholarship dedicated to advancing intervention 
research in light of the low publication rates. 

In the present study, the frequency of published articles examining the effectiveness of aca- 
demic interventions using a single-case design was variable across the 11-year period from 2003 
through 2013 with greater frequency in 2006 and 2010 and a steady downward trend from 2010 to 
2013. This result is consistent with and extends the findings of the literature review conducted by 
Villarreal and associates (2013), who reported a significant downward trend in single-case inter- 
vention research published in six school psychology journals from 2005 to 2009. This variability 
and general decline in the frequency of single-case design intervention research is surprising given 
the emphasis on using single-case design research within a Response to Intervention framework 
for meeting students’ needs (Jimerson, Burns, & VanDerHeyden, 2007; Riley-Tillman & Burns, 
2009) and for evaluating evidence-based practices and professional accountability among school 
psychologists (Morrison, 2013; Morrison, Graden, & Barnett, 2009). 

Despite calls from intervention researchers to examine impact in terms of effectiveness and 
the amount of instruction/intervention time to achieve effectiveness (efficiency), the results of this 
study indicate that efficiency variables remain largely absent in the academic intervention rese- 
arch, with rarely more than two efficiency studies published in a given year. In the present review 
of the literature, 14 articles were identified that included a calculation of efficiency. The literature 
review conducted by Bramlett et al. (2010) identified only two articles that included the amount of 
instructional time needed to generate the effect obtained among publications from 1995 to 2005. 
This slight increase is negligible given the larger proportion of intervention effectiveness studies 
that did not include an efficiency variable. Furthermore, this finding was particularly disconcerting 
given that the amount of instructional/intervention time was reported in minutes for slightly more 
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than half (47, or 55.3%) of the 85 articles included in the present review and presumably a learning 
rate could have been calculated with relative ease. 

Several limitations of the present study are worthy of consideration when examining its fin- 
dings. Although the journals examined represent major publication sources in school psychology, 
they do not represent all of the possible publication outlets in school psychology, which would 
also include School Psychology International and the Journal of Applied School Psychology. By in- 
cluding two journals relevant to school psychology that are more inclined to publish research with 
a behavioral theoretical orientation, the authors of this study sought to maximize the likelihood of 
finding academic intervention research using single-case designs. A different selection of journals 
may have yielded a markedly different sample of articles. It is also possible, as Villarreal et al. (2013) 
point out, many school psychology researchers may publish their intervention studies in journals 
related to their particular research focus (e.g., early childhood intervention, autism). Future research 
should explore the frequency of intervention research published by school psychology-affiliated 
researchers in journals not exclusively relevant to school psychology. 

The body of research on empirically-validated academic interventions was restricted in this 
study by focusing only on studies that used single-subject design methodology. As such, the fre- 
quency of academic research studies that included effectiveness and efficiency outcome variables 
may have been greater than reported in this study if other research designs were included in the 
literature review. Furthermore, the body of intervention research literature was further limited by 
focusing on interventions targeting academic skill remediation. Future research should explore 
the degree to which the findings of this study hold when the focus is on interventions targeting 
social/behavioral concerns. 


Conclusions 


The current demand for accountability and evidence-based practices within the field of school 
psychology creates an undeniable sense of urgency among for researchers and practitioners to use 
empirically-validated studies as a basis for intervention design and selection. The findings of this 
study indicate that the current status of intervention research targeting academic interventions is 
not as extensive as it needs to be to inform research and practice in school psychology and largely 
fails to consider intervention efficiency along with intervention effectiveness when evaluating the 
impact of the intervention. The need for more academic intervention research using single-case 
designs and reporting intervention effectiveness and efficiency persists as a critical problem within 
school psychology. 
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Appendix 


CODING PROTOCOL 


The Coding Protocol was created using a Microsoft Excel with the following variables: 


Variable 


Coder 


Source 


Year 
Citation 
Participants 


Setting 


Length 


Duration 


Design 


Effectiveness 


Description 


This variable identifies the 
coder 


This variable identifies the 
journal 


This variable identifies the 
publication year 


This variable uniquely 
identifies the article 


This variable describes the 
participants of the study 


This variable describes 
the location in which 
the intervention was 
implemented 


This variable identifies 
the number of student 
participants in the study 


This variable identifies 
the length of intervention 
condition 


This variable identifies the 
duration of the planned 
intervention session 


This variable identifies the 
single-case design 


This variable identifies the 
effectiveness variable(s) 


Coding 


Enter the coder’s initials 


Enter the journal’s abbreviation 

JEBP = Journal of Evidence Based Practice in Schools 
JOBE = Journal of Behavioral Education 

JSP = Journal of School Psychology 

PITS = Psychology in the Schools 

SPQ = School Psychology Quarterly 

SPR = School Psychology Review 


Enter the four-digit year. If the on-line publication year preceeds the 
hard copy publication year, enter the hard copy publication year. 


Enter authors last names and publication year 


Enter grade, age, gender, race, general ed/special ed status, and 
disability (if applicable) for each participant 


Enter description provided by the authors of the study. 
Examples include: classroom or separate room near the classroom 


Enter the number of students who participated in the study. If the 
study used peers as interventionist (e.g., peer tutoring), record only 
the number of target students who received support. 


Enter description provided by the authors of the study regarding the 
number of days or weeks of intervention. If multiple interventions 
were implemented or modifications were made to the intervention 
package, record the total length of the intervention condition(s). 


Enter the duration of each intervention session in minutes. This is 
typically reported as a range. Example: 10-12 minute session 


Enter the code for the design as: 

1 = Multiple-baseline design 

2 = Alternating treatment design/Multielement design 
3 = ABA or ABAB design 

4 = Multiple probe across problem set 

5 = Other (Specify) 


Enter description provided by the authors of the study regarding 
their dependent variable(s) 


Efficiency 
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This variables identifies the | Enter description provided by the authors of the study regarding 
efficiency variable 
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their measure of efficiency — intervention effect per instructional/ 
intervention time in minutes. Learning rates, rates of improvement, 
and student growth per week are not efficiency measures if they are 
based on the passage of time (i.e., days, weeks) and not anchored 
to instructional/intervention time. 
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